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ABSTRACT 

This thesis uses pre-commissioning academic and demographic factors, along with 
flight school performance data to measure pilot success in flight school. The goal is to 
determine if undergraduate major or sch «i attended affect flight school performance. 
Measures of effectiveness include: (1) Flu; School Completion Status, (2) Aviation Pre- 
Flight Indoctrination Composite Scores, a: —} Primary Flight Training Composite Scores. 
Recruitment for naval aviators is focused. dividuals with “technical majors,” according 
to present policy of the Naval Recruiting | mnmand. This recruiting philosophy is based 
on the “Rickover Hypothesis,” which postuiates that naval officers with technical degrees 
are superior to naval officers with non-technical degrees. The Logit model showed that 
aviators with engineering degrees have a statistically greater chance of completing flight 
school than aviators with non-engineering technical or non-technical degrees. In addition, 
the results showed an association between academic background and flight school 


performance. This research justifies the current Navy policy of concentrating aviator 


recruitment efforts on individuals with technical degrees. 
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EXECUTVE SUMMARY 

The Naval Recruiting Command aviator recruiting policy is to obtain college 
graduates who possess engineering and science degrees. This recruiting philosophy is 
based on the “Rickover Hypothesis,” which postulates that naval officers with technical 
degrees are superior to naval officers with non-technical degrees. Bowman (1990) 
examined this hypothesis for officers und2rgoing nuclear training and found that technical 
degrees significantly increased the proba‘ ‘lity of completing this training, but that little 
effect on subsequent performance and promotion was observed. The current study 
examines the Rickover hypothesis for student aviators by comparing their academic and 
flight performance during training. These comparisons are made based on flight school 
completion/attrition status and on composite scores attained during Aviation Pre-flight 
Indoctrination and Primary flight training. It is expected that differences in performance 
will be found based upon undergraduate major or college attended. In addition, differences 
in performance will be noted based upon scores received on the aviation selection test 
battery. Individuals who have a higher Academic Qualification Rating and/or a higher 
Flight Aptitude Rating will perform better during the academic and flight portions of 
training. 

Using Classification and Regression Trees, Logistic regression and Least-Squares 
regression, the performance of student aviators in the aviation-training pipeline (measured 
by completion or attrition, and by flight school grades) 1s modeled. Demographic variables 
from the officer master file, along with aviation selection test battery performance data, are 


used as predictors to measure success in flight school. These factors include sex, race, 


XV 


ethnicity and commissioning source, along with undergraduate major, college attended, and 
undergraduate academic performance grades. 

Classification Tree and Logit models are developed to predict individual success in 
completing training. Results indicate that academic major is a significant predictor of 
flight school completion, along with college, race, Aviation Oiahicoton Rating and Flight 
Aptitude Rating. Engineers are more likely.t: complete flight school than individuals who 
hold non-engineering technical or non-technical degrees. Further, graduates from the 
Naval Academy have a significantly higher completion rate than individuals who attended 
other colleges. 

Regression Tree and Least-Squares regression models are developed to measure 
relative success of individuals throughout the flight training pipeline by using predictors to 
model their academic and flight performance grades during Aviation Pre-flight 
Indoctrination and Primary flight training. Only individuals who successfully completed 
flight school are considered. Significant predictor variables include Academic 
Qualification Rating, Flight Aptitude Rating, race, ethnicity, college and major, indicating 
that each predictor has an influence on the composite score attained while attending flight 
school. . 

In summary, there is evidence to suggest that academic major and college attended 
affect performance in flight school. These factors, along with race, ethnicity, Academic 
Qualification Rating and Flight Aptitude Rating can be used by the recruiting command to 
screen potential applicants using the models developed. Individuals can be compared, 
based on their individual characteristics, to determine the best candidates for acceptance 


into flight school. 
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I. INTRODUCTION 
A. BACKGROUND 

As the United States makes the transition into the 21" century, changes in the 
recruiting and selection process for naval aviators could bring an improvement to the 
Navy's return on investment. The qualification process for aviators is intense, with many 
applicants competing for a limited number of billets. Many factors contribute to selection 
failure. First among them are stringent initial Seen requirements. The applicant 
screening process requires all individuals to pass both an Aviation Selection Test Battery 
(ASTB) and a military physical exam. These initial qualification standards result in a 
disqualification rate of 75% from the initial applicant pool (Williams, Albert, and Blower, 
1999). Further, individuals must then apply for acceptance into the flight-training program, 
whereupon successful candidates are ordered to flight school for initial and follow-on flight 
training. Each stage in the qualification process brings about a reduction in the pool of 
_ potential aviators, resulting in 1 fleet-qualified aviator for about every 25 applicants (Wahl, 
1998). 

The aviator recruiting effort is focused on individuals with “technical majors,” 
according to present policy of the Naval Recruiting Command (NRC, 1999). This idea is 
in line with the “Rickover Hypothesis,” which presumes that individuals with technical 
degrees are better prepared and make better naval officers than individuals with non- 
technical degrees. However, Bowman (1990) found a very weak statistical relationship 
between USNA major and fleet performance, using fitness reports and other job 
performance variables. In addition, he argued that the need for technical skills diminishes 


as officers advance to positions requiring greater managerial and administrative skills 


] 


(Bowman, 1990). Bowman found that although a technical degree significantly affects the 
likelihood of completing the nuclear training pipeline, little effect on performance and 
promotion is apparent once individuals complete training and enter the fleet (Bowman, 
1990). Performance in areas not directly associated with the formal training process, such 
as leadership, management, and interpersonal! skills, affect overall individual performance 
and promotion rates, yet have little to do with academic background. Thus, by limiting the 
potential applicant pool to individuals with technical degrees, a significant portion of the 
population of potential aviators 1s removed. from consideration. These are individuals who 
lack a technical degree but might otherwise pass screening requirements, flight school and 
thus, become Naval aviators. 
B. RESEARCH OBJECTIVES 

This study examines the effect of academic background on pilot performance 
within the flight-training pipeline. Each analysis focuses on the effect of academic 
achievement as measured by grades, undergraduate major, and the isin between 
college quality, commissioning source and flight school performance. Comparison 
between technical and non-technical aviators is made to determine if pilots with technical 
degrees perform better through the flight-training pipeline. 

A second objective was to analyze the effect of undergraduate major on retention 
and promotion rates of pilots who completed the flight-training pipeline during the 1980’s, 
based upon their flight school performance, and in conjunction with other demographic 


variables. This, however, was impossible because of incomplete data on flight school 


aviator performance in years previous to 1990. 








c. SCOPE AND LIMITATIONS 

The focus of this study is on the effects of undergraduate academic major and 
college attended on student aviator performance. Effects of other college characteristics, 
such as participation in athletics or other extracurricular activities, are not included. This 
intention is to test the validity of the “Ric kover Hypothesis” with regard to the aviation 
community and to provide recruiters wit’ ‘nformation that will allow them to focus on 
those college students who are likely ~ perform well in the aviation community. 
Academit major and college are included . the models as independent variables. Flight 
Aptitude Rating (FAR) and Acalenic Qualification Rating (AQR) are also included since 
they are designed to predict performance in flight school. Demographic variables such as 
sex, race, ethnicity and commissioning source are included to control for possible 
confounding. (For example, if it were the case that black aviators both performed better on 
average in flight school and were more likely to have technical degrees than non-blacks, 
the effect of having a technical degree would be over-stated in a model that did not include 
race as an independent variable.) 

This study is limited to pilots who graduated from flight school between 1990 and 
1999. Individuals from the United States Naval Academy (USNA), Naval Reserve Officer 
Training Units (NROTC) or from Officer Candidate School (OCS) are included. The data 
set does not include those who transferred into the aviation community from another 
community or from another service. Officers with prior enlisted service are omitted if they 
served greater than four years in an enlisted capacity. 

The most recent revision to the ASTB occurred in 1992. Due to the changeover 
(further described in Chapter II), it is not yet possible to consider the long-term retention 
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and promotion characteristics of officers who have qualified under the present applicant 
screening guidelines. Because it takes up to two years from initial test taking to 
commissioning, and a further two years for individuals to complete their flight training, 
these officers could only recently have started their obligated service requirement and 
would not have yet reached time-in-grade requirements for Lieutenant Commander. In 
fact, most will not have yet completed their initial service requirement. Therefore, it is 


difficult to predict retention likelihood for these individuals from results found in this 


study. 
D. ORGANIZATION 

This study is organized into five chapters. Following the introduction and 
background contained in Chapter I, Chapter II reviews previous studies and literature 
related to this area of research. Chapter JII describes the data files that are employed for 
this research. A detailed description of model specifications and an overview of 
methodology are given, with an explanation of the dependent and independent variables. 
Chapter IV presents the results from this study using Classification and Regression Trees 


(CART), Logistic regression and Least-Squares regression. Chapter V offers conclusions 


and recommendations based on the results of the previous chapter. 














Il. LITERATURE REVIEW 

A. QUALITIES OF AN AVIATOR 

Many studies have been conducted that examine the characteristics of “good” 
aviators; however most have focused on the applicant selection process, rather than 
measuring skill and performance over the long term. According to the Federal Aviation 
Administration, “... no one is born a nati: al pilot. Competent commercial pilots become 
so through study, hard work and experic «e” (U.S. Department of Transportation, p. 1, 
1995). In order to fly safely, all aviato:; must master basic airmanship, operation of 
aircraft systems, and navigation. Commercial pilot tests cover a wide range of subject 
areas, in addition to specific instruction in the aircraft category for which rating is sought. 
Military aviators must also master these functional areas, in addition to specialized 
requirements necessary to work in a combat role. They must also understand the operation 
of weapon systems and cope with high stresses imposed by operation in combat 
environments. A list of knowledge areas from the 1995 Commercial Pilot Knowledge Test 
Guide is given below as suggestions of topics to be covered by rating exams (U. S. 
Department of Transportation, 1995): — 


1. The Federal Aviation Regulations that apply to commercial pilot privileges, 
limitations, and flight operations. 


2. Accident reporting requirements of the National Transportation Safety Board. 

3. Basic aerodynamics and the principles of flight. 

4. Meteorology, including recognition of critical weather situations, wind shear 
recognition and avoidance, and the use of aeronautical weather reports and 


forecasts. 


5. The safe and efficient operation of aircraft. 


6. Weight and balance computation. 
7. Use of performance charts. 
8. Significance and effects of exceeding aircraft performance limitations. 


9. Use of aeronautical charts and magnetic compass for pilotage and dead 
reckoning. 


10. Use of air navigation facilities. 

11. Aeronautical decision-making and judgment. 

12. Principles and functions of aircraft systems. 

13. Maneuvers, procedures, and Bae sency operations appropriate to the aircraft. 


14. Night and high-altitude operations. 


15. Descriptions of and procedures for operating within the National Airspace 
System. 


In addition to the above areas, naval aviators must also master the complexities of 
combat weapon systems and the effects each has when carried and launched. They must 
understand launch parameters of the weapons, their flight characteristics, and the additional 
system controls they require. In another area, most naval aviators are faced with the 
obstacle of landing an aircraft on an unstable platform (ship or aircraft carrier), limited in 
size, and moving relative to the aircraft. These additional factors are summed up by 
Pohlman & Fletcher (1999), who state that military pilots must (Pohlman & Fletcher, 
1999): 


1. Plan the route through space in relation to the intended target, suspected threats, 
actual threats, other known aircraft, wingmen, and weapons. 


2. Monitor the aircraft display for electronic notification of threats. 


3. Differentiate among threat displays (these can portray 15 or more different 
threats). 





4. Plan ingress to and egress from the target. 


sn 


Set switches for specific missions during specific periods of the flight. 

6. Monitor radio chatter on multiple frequencies for new orders and threat 

notification. 

7. Monitor progress along the planned route. 

8. Calculate course, altitude, and ‘'rspeed corrections. 

9. Plan evasive maneuvers for ea: -: type of threat and position during the mission. 

10. Plan weapons delivery. 

In short, the demands of military (and commercial) flying require the ability to 
make quick mental adjustments using proper judgment in response to rapidly changing 
situations. Because of this necessity, assessment procedures to determine qualified 
candidates for aviator training are rigorous. Proper candidate selection saves time, material 
and funding, and results in improved quality and operational readiness. Because of the 
expense and complexity required to train competent pilots, “almost every test in the 
psychological arsenal has been evaluated at one time or another to determine its 
applicability for aircrew selection” (Hunter, p. 129, 1989). In addition, Hilton and Dolgin 
wrote that ee may be no other “occupation in the world that benefits more from 
Seeoniel selection technology than that of military pilot” (Hilton & Dolgin, p. 81, 1991). 

According to Pohlman and Fletcher (1999), three conclusions may be drawn from 
review of studies related to aviator selection. The first is that nearly all validation studies 
conducted concern the ability to predict performance in training, rather than long-term pilot 
performance. Inherent in this reasoning is the belief that an aviator who does well in 


training will continue with good performance throughout his or her career. Training 


validation is good because it identifies potential failures early, resulting in decreased costs 
borne by the government or individual trainee. According to Hunter, flight training is “the 
most expensive of the many training programs conducted by the military services” (Hunter, 
p. 129, 1989). The second conclusion is that there is little relationship between general 
intelligence and aviator performance. It was noted that newer tests of mental ability might 
better identify aspects of general intelligence that predict aviator performance. Finally, 
Hunter (1989) found that of the 36 studies conducted, “only those concerned with 
instrument comprehension and mechanical comprehension were consistent predictors of 
success.” In a more recent study by Hunter and Burke (1995), it was found that the best 
correlates of success in training were sample tests of job performance, gross dexterity, 
mechanical understanding, and reaction time. General ability, quantitative ability, and 
education were again found to be poor correlates of success (Hunter & Burke, 1995). 
B. NAVAL AVIATION SELECTION TESTS 

Predictors of success in flight training were developed during the Second World 
War. Large numbers of naval aviators were needed to meet the needs of the United States 
war effort, and a selection process for potential candidates was introduced. Subsequent 
revision and examination led to the formulation of the first naval aviation selection test, 
called the Aviation Selection Test Battery (ASTB) (ea: 1996). This test, revised in 
1953, and again in 1971, was composed of two parts: an Academic Qualification Test 
(AQT) and a Flight Aptitude Rating (FAR). The test was used as a measure to determine 
potential applicant success in flight training, using biographical data derived from a 
questionnaire that asked about family background, personal and medical history, 


environmental influences, education and vocational interests, in addition to academic skills. 
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The AQT portion was a general intelligence instrument designed to predict performance in 
the academic phase of training. The FAR was a composite score based upon individual 
scores of a Mechanical Comprehension Test (MCT), a Biographical Inventory (BJ) and a 
Spatial Apperception Test (SAT). This composite score was intended to predict the 
probability of a student’s success in the flight portion of training. In 1992, the test was 
again replaced in favor of a newer version. 

The 1992 ASTB is the most recent revision in a series of cognitive tests used as a 
selection measure for potential naval aviators. This revision took place due to demographic 
changes, decreases in predictive validity of the previous test, possible AQT/FAR test 
compromise, and finally, due to changes in federal guidelines regarding employee selection 
procedures (Frank & Baisden, 1993). The new version of the ASTB consists of six sub- 
tests: 1) Math-Verbal Test (MVT), which tests general intelligence; 2) Mechanical 
Comprehension Test (MCT) which test the ability to perceive physical relationships and 
solve practical problems in mechanics; 3) Spatial Apperception Test (SAT), which tests the 
ability to perceive spatial relationships from different orientations; 4) Aviation and 
Nautical Test (AN), which tests for general aviation and nautical knowledge; 5) 
Biographical Inventory (BI) which is a questionnaire of personal indi and interests; and 
6) Aviation Interest (AI) which is a questionnaire of aviation-related items. Weighted 
combinations of the sub-tests result in three separate scores, each on a 9-point scale. These 
scores are used to predict attrition, academic performance, and basic flight performance of 
potential aviator candidates. The first score, called the Academic Qualification Rating 
(AQR), predicts flight school academic performance. The Flight Aptitude Rating (FAR) 
predicts basic flight performance while the Biographical Inventory (BI) predicts attrition. 
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The new version of the ASTB differs from the previous tests (utilizing AQT/FAR scores) 
in that only the BI is intended to predict attrition from primary flight training. Navy 
applicants must obtain qualifying scores of 3, 4 and 4 on the AQR, FAR, and BI for initial 
consideration into the flight program. Candidates receiving qualifying scores are further 
screened prior to acceptance into the training pipeline. 
C; THE NAVAL FLIGHT TRAINING PROGRAM 

Candidates selected for naval aviation training first attend a six-week course of 
ground schoo! training at Aviation Preflight Indoctrination (API). During API, students 
master topics such as aerodynamics, aircraft engines, air navigation, meteorology, flight 
rules and regulations (FR&R), physical conditioning, and water survival, resulting in a total 
of 231 hours of instruction being received by each candidate. Upon completion of API, 
candidates are split into pilot and Naval Flight Officer (NFO) programs, and proceed to 
their respective training pipelines (Williams, Albert & Blower, 1999). ee flight 
training introduces the student to actual flying experience, including basic instrument and 
radio instrument familiarization, close formation and night flying exercises. All flights are 
flown either in actual T-34C Turbo Mentor aircraft or the T-34C flight simulator. Student 
aviators. are graded on preflight knowledge, emergency procedures, ability to think and 
react under stress, and other items related to the particular flight mission. Each student 
receives a grade between 1.0, which is considered “unsatisfactory,” and 4.0, considered 
“above average,” respectively. There are 530 graded items completed by each student 
aviator. A final score ranging from 1.0 to 4.0 is assigned, based upon the average of all 
scores received. After successful completion of primary flight training, each student then 
enters one of four aircraft pipelines for intermediate flight training. Selection to a 
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particular pipeline is based on flight grades and the current needs of the Navy. In general, 
however, those with highest grades are selected for jets, followed by carrier-based propeller 
aircraft, maritime propeller aircraft and helicopters (Wahl, 1998). 

Naval aviators are disqualified or dropped from the training pipeline for a variety of 
reasons. As stated above, many potential applicants are disqualified during the initial 
recruitment and selection phase; however, significant numbers drop out during the training 
process. A large cost is incurred when : .dividuals drop from the training pipeline. As 
applicants progress through the training process, the cost for each applicant increases 
significantly. Reasons for withdrawal from training vary significantly, with categories 
including Drop on Request (DOR), Flight Failure, Not Physically Qualified (NPQ), Not 
Officer Material (NOM), Not Aeronautically Adaptable (NAA), Academic Failure and 
Other (misconduct, etc). Approximately 60% of all candidates fail due to medical 
disqualification. The Navy considers this amount to be unavoidable and not preventable. 
The goal for the Navy is to limit the number applicants withdrawing from training due to 
academic, flight failure, or DOR. Identification and early intervention by the Navy may 
result in subsequent cost savings to the training command and the Navy as a whole. 

D. OFFICER PERFORMANCE MEASUREMENT 

In 1976, ADM Hyman G. Rickover, Director of Naval Reactors, testified before the 
House Armed Services Committee that teaching Management as a major subject for an 
undergraduate did not contribute to the ability af a junior officer to do his job. He believed 
that all midshipmen should be taught electives limited to the technical sciences and that 
social sciences should be specifically excluded (Hearings on Military Posture, 1976). This 
belief led to the “Rickover Hypothesis,” that the best naval officers are those who have a 


1] 


technical undergraduate major (Bowman, 1990). Bowman studied USNA graduates from 
1976 through 1980 who entered the Surface and Submarine officer communities. He found 
that technical expertise diminished as officers advance to positions requiring greater 
managerial and administrative skills (Bowman, 1990). In addition, he found that retention 
factors are based upon personal characteristics, including perceived monetary options near 
the end of one’s obligation, rather than academic background. Bowman’s study applied 
only to USNA graduates, rather than the whole population of officers in the communities 
he studied. 

Other studies that measure naval officer performance in a wide range of settings 
have been conducted. These examine topics ranging from specific aviator primary flight 
suconnante of USNA graduates (Reinhart, 1998), to Surface Warfare Officer (SWO) 
performance as related to commissioning source, undergraduate education and Navy 
training (Nolan, 1993). The following paragraphs highlight the research and results in 
areas related to officer productivity. 

Reinhart (1998) utilized ASTB scores, along with other demographic variables, to 
compare primary flight training performance of 1995 and 1996 USNA graduates. He 
found that individuals who scored higher on the BI were more likely to complete primary 
flight training than those with lower scores. In addition, he found that those with higher 
AQR and FAR scores achieved higher grades than individuals with lower scores. 

Foster (1990) measured the relative productivity of SWO and Submarine Officer 
(SO) officers from different accession sources, using a performance index derived from 


aspects of officer fitness reports, which allowed officers to be ranked or compared with one 
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overall grade. Foster found that USNA graduates were rated higher, and that by a small 
margin, they were promoted earlier than graduates of other commissioning sources. 

Nolan (1993) conducted a study of SWO promotion, retention and qualifications, as 
related to commissioning source, education and training. He used “Barron’s Profile of 
American Colleges” to rank colleges according to their competitiveness. In addition, he 
utilized the Academic Profile Code (APC) that is assigned to each officer and which 
summarizes portions of an individual’s undergraduate academic performance. The APC is 
broken down into three sections: grade point average (GPA); mathematics qualification 
code (MQC); and the technical qualification code (TQC). His results suggest that 
attendance at higher-rated colleges, having higher educational quality, was positively 
correlated to higher performance measures of effectiveness, such as promotion and early 
attainment of qualifications. 

Woelper (1998) measured the impact of college grades, undergraduate major, 
college quality and commissioning source on SO job performance, as evidenced by early 
promotion and retention. His intent again was to test the “Rickover Hypothesis” that a 
strong technical background makes better naval officers. He found that engineering majors 
have higher completion rates through the training pipeline; however, major had an 
insignificant effect on junior officer performance or promotion to LCDR (Woelper, 1998). 
E. BACKGROUND CONCLUSIONS 

Aviator performance is difficult to measure. However, much work has been done 
to identify individual traits that point to successful completion of flight training. The 
civilian literature cited above (Puhlman & Fletcher, 1999) indicates that primary in 
importance is an individual’s ability to master the mechanics and coordination required for 
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flying. The ASTB enables the U. S. Navy to screen potential applicants, prior to 
committing limited funding and resources to individuals not suited for aviation. Further, 
Aviation Pre-flight Indoctrination attempts to identify as early as possible those individuals 
who are unable to complete the training regimen. Research conducted on the SO 
community seems to indicate that individuals with technical degrees are better than those 
with non-technical degrees, at least during. training (Woelper, 1998). Work has been 
conducted that indicates that while academic background has a positive influence on officer 
performance at the initial training point, the lasting effect of undergraduate experience 
diminishes as time passes (Woelper, 1998). From the previous literature reviewed, it is 
expected that an aviator’s academic background or college attended will influence flight 
school performance. In addition, higher ASTB scores will correlate with superior flight 
school performance, as evidenced by previous studies that have validated this test 


(Reinhart, 1998, Williams, Albert & Blower, 1999). 
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lil. DATA AND METHODOLOGY 

A. DATABASE DESCRIPTION 

The first data set, provided by the Defense Manpower Data Center (DMDC), was 
derived from the Officer Master File (OMF), and contained both pre-commissioning and 
post-commissioning characteristics of naval officers. Data include demographic, 
educational background, billet assignment and promotion factors. The second data set 
contained information on flight school selection and performance by each individual. This 
data set was obtained from the Naval Aerospace Medical Research Laboratory (NAMRL) 
and contained 8,882 observations of students who were admitted for training at Naval 
Aviation Schools Command in Pensacola from 1988 through 1999. Included in this set 
were officers from other service branches and foreign countries. All non-USN naval 
aviators were removed from the data set, reducing the overall number to 6115 members. 
This file was then merged with the OMF file, resulting in 5123 matches in the combined 
data set. The resulting aviator subset of data was further reduced in size to 3937 
observations due to missing observations of other key variables, including undergraduate 
major, aviation pipeline, flight school performance scores and university attended. This 
data set was further divided by removing the NFO’s from consideration, leaving 2612 
student aviators. Once the final data set was completed, variables were created or factors 
modified to better isolate characteristics of interest to this study. A breakdown of the final 
data set column descriptions can be found in Appendix A. 
B. KEY INDEPENDENT VARIABLE DESCRIPTION 

Several key variables Wer created from the above data sets. The first variable 
derived was “MAJOR,” which separated college degrees awarded into three categories. 


15 





These categories were obtained from the Office of the Chief of Naval Operations (CNO) 
which listed “Non-Technical,” “Technical” and “Math-Science Technical” majors as 
possible courses of study for the future naval officer (CNO, 1999). Non-technical majors 
such as history and accounting were assigned to the first factor level, while non- 
engineering technical majors such as math and physics were assigned to the second level. 
Engineers were assigned to the third factor level. As indicated above, if information about 
the college degree awarded to an individual was absent from the data set, then the 
individual was removed from consideration. 

The second key variable is “COLLEGE,” which was created using a combined 
listing of collegiate rankings as provided by U.S. News and World Report (Elfin, 1990; 
1995). Colleges were rated according to six sibiiiaael — reputation, selectivity, faculty 
resources and financial resources, retention and alumni satisfaction. The schools were 
divided into “National” and “Liberal Arts” colleges. They were then ranked according to 
the attributes listed above. The top quartile schools by rank were included for the purposes 
of this study. In order to do this, the top schools from both 1990 and 1995 were merged to 
provide coverage for aviators who otherwise might not have been included. Each 
individual was coded as having attended either “none,” “national” or “liberal” to signify the 
school rank they attended. After final review, the factor levels for “liberal” and “national” 
were combined because only a few individuals attended top ranked liberal arts colleges 
included in the data set. None of the service academies were included in the final rankings 
by U.S. News & World Report even though a significant percentage of individuals attended 


the Naval Academy. These individuals were placed within their own factor level. 
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“Academic Profile code” (APC) is another key variable from which three sub- 
categories were obtained. The APC code 1s a three-digit code that summarizes pertinent 
portions of an Officer’s prior college performance. The first digit indicates overall 
academic performance. The second represents mathematical background and the third 
represent course coverage in science and technical fields. The three separate digits reflect 
an individual’s cumulative grade point average, exposure to and performance in calculus- 
related mathematics and selected science/c gineering areas. Use of the APC is limited in 
that only about 40% of the pilots in the data set have been assigned this code. The 
individual digits of the APC code were re-classified into a binary variable based upon 
individual performance. A binary code of 1 was assigned to individuals who demonstrated 
B+ or better GPA, math and/or technical backgrounds. 

Other variables were derived foi the data set for ease in manipulation. “RACE” 
and “ETHNIC” categorical variables were coded according to major divisions. Initially, 
they consisted of numerous factors, some of which contained only a few individuals. 
“RACE” was reduced to factors of “White,” “Black,” and “Other”. “ETHNIC” was coded 
to allow “None,” “Hisp,” “Asian” and “Other” as possible factor levels. Commissioning 
Source was also used, with factor levels of “OCS,” “NROTC,” and “USNA” being created 
for the categorical variable “SOURCE.” Finally, both “FAR” and “AQT” integer variables 
were used from results of the ASTB scores. 

C. KEY DEPENDENT VARIABLE DESCRIPTION 
Aviator’s flight school completion status was determined by “CURRSTAT,” which 


identified whether or not an individual completed or withdrew from flight school. This 





categorical variable allows model formulation using either classification trees or by Logit 
regression due to its binary nature. 

Another measure of effectiveness was the composite score obtained from 
attendance at the initial flight training school attended. This score, from the Navy Aviation 
Schools Command (NASC), is based upon a student’s performance in API and measures 
academic performance in the early stages of the aviation-training pipeline. This 
explanatory variable was called “NASCNSS” and is continuous in nature. 

Two other continuous variables were used. The first, “PASS,” reflected the 
standardized academic score achieved during Primary flight training, while “PFSS” was a 
measure of flight performance during the same period. 

D. METHODOLOGY 

Statistical analysis starts by using Classification and Regression Trees (CART) to 
model performance of individuals in the aviator-training program. Trees have the 
advantage that they are easier to interpret when the predictors are a mixture of categorical 
levels and numeric entries. Furthermore, the response can either be a continuous or a 
categorical value. Thus CART can be used for either “CURRSTAT,” which is a 
categorical response, or for the other response variables, which are continuous. 

A tree is formed by measuring the amount of variation or deviance between each 
predictor and the response variable. The predictor that reduces variation or deviance by the 
greatest amount becomes a “parent” node and a partition is made, dividing the parent node 
into two “child” nodes that contain all the other predictor variables. Each child node then 
becomes a parent and the process repeats itself. This process continues until there are no 


members of a factor left to split or if a preset minimum deviance reduction level is reached. 
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The process is binary because parent nodes are always split into exactly two child nodes 
and recursive because the process can be repeated by treating each child node as a parent. 
(Breiman, et. al., 1984). 

After the initial tree model is determined, cross-validation (CV) is used to optimize 
the predictive reliability of the tree model. Several methods of cross-validation exist; 
however this thesis utilizes V-fold cross-validation (Statsoft, 2000). This type of cross- 
validation is useful when no test sample is «:vailable and the learning sample is too small to 
have the test sample taken from it. A specified V value for V-fold cross-validation 
determines the number of random sub-samples, as equal in size as possible, that are formed 
from the learning sample. The classification tree is computed V times, each time leaving 
out one of the sub-samples from the computations, and using that sub-sample as a test set, 
so that each sub-sample is used V —1 times in the learning sample and just once as the test 
sample. 

Finally, the total amount of deviance reduction is measured as each node of the 
crass-validated tree is generated. As the tree increases in size and complexity, an optimal 
point will be reached. At this point, the most important splits in the tree have been 
determined. The tree is then pruned so that only these factors are used in final tree 
construction. The resultant tree provides the user a tool for making predictions about 
individuals contained in the data set. Further, the tree has effectively screened and 
identified the most useful predictor variables from the data. These variables provide the 
initial inputs to the Logistic and Least-Squares regression models subsequently formulated. 
Since the focus of this study is to determine the impact on academic major and quality of 
college attended, “COLLEGE” and “MAJOR” are added to the Logistic and Least-Squares 
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models along with the predictors identified by CART, if not already identified by the tree 
created. 
The Logistic model uses the predictor variables identified by the CART model, 


along with the academic background variables to get an estimated “Logit” for individuals 


in the data set. This value is analogous to the estimator Y that is calculated using Least- 
Squares regression. A full model is developed, and then a step function is applied to 
reduce this model, keeping only those predictors that provide the most information in the 
model. This step function uses the Akaike Information Criterion (AIC), and measures the 
model effectiveness, based upon sample size and the number of predictors in the model. If 
a predictor does not provide statistically significant information based on the AIC, it is not 
included in the final model formulation (Agresti, 1990). The final Logistic model can be 


formulated as: 
L,=f\+ PX, +hXi ++ BX 
Using an inverse transformation, a predicted probability of flight school success for 


the individual can be calculated. Thus, 


- 
P — Ste ar 
l+e 
Using this transformation, the predicted success probability for each member of the data set 


can be computed, based upon his or her educational and demographic background and 


flight school performance data (Hamilton, 1992). 
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IV. RESULTS 
A. DATA ANALYSIS 

The student aviator data set contains 2612 members, of which 150 are women. Of 
these members, 2144 completed training, while 468 failed to complete the pipeline. The 
data set examined did not provide an explanation for early departure from the flight- 
training program. There were 1235 NROTC graduates, 991 USNA graduates, and 386 
OCS graduates in the program. A complete breakdown of these and all other elements of 
the student aviator data set can be found in Appendix B. 

A two-way contingency table was arranged and a chi-square test of independence 
was conducted, with a null hypothesis was that there were no differences between majors 
based upon the proportions that completed the flight-training program. The null hypothesis 
was rejected, indicating that there is a difference between majors (chi-square = 29.9662, df 
= 2, p-value = 0). Comparisons were made to determine how the majors differed from 
each other. No significant differences between non-technical and technical majors were 
found (chi-square = 1.0226, df = 1, p-value = 0.3119). However, between non-technical 
majors and engineers, significant differences exist (chi-square = 29.3487, df= 1, p-value = 
0). Technical majors and engineers were compared, with significant differences found 
(chi-square = 6.3579, df = 1, p-value = 0.0117). Finally, technical majors and engineers 
were eenabined and then compared to non-technical majors, resulting in significant 
differences being detected (chi-square = 23.8068, df = 1, p-value = 0). In summary, non- 
technical majors have significantly lower completion rates than the combined set of 


technical and engineering counterparts (Table 1). 


21 









Non- Technical Engineering | Technical Total 
Technical Majors Majors and 
Majors Engineers 


1036 1108 aap 
1321 1291 2612 


Proportion 
Completing 78.43% 81.25% 87.23% 85.82% 82.08% 
Training 


Table 1. Student aviator completion rate, based upon Undergraduate Major. There is no 
evidence that differences exist between non-technical and technical majors, but do exist 
with engineering majors, and between non-technical majors when compared against both 
technical majors and engineers combined. _ 










A CART was formed using the “CURRSTAT” dependent variable, indicating 
completion/attrition status. The model is as follows: 
CURRSTAT ~ SEX + AQT + FAR + RACE + MAJOR + ETHNIC + SOURCE + 
COLLEGE + GPA + MATH + PHYSICS 

A full, saturated tree was generated and cross-validated using V-fold CV with ten 


holdout sets. Figure 1 displays the deviance plot of the cross-validated tree. 


65.000 5.500 3.600 3.000 2.300 1500 0930 0.480 
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Figure 1. Size vs. deviance plot for 
CURRSTAT model. 
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The minimal point of the deviance plot 1s five, indicating that a tree with five terminal 
nodes provides the most accurate prediction. The tree was pruned to reflect this optimal 
level. All variables were made available to the algorithm; Figure 2 shows that only FAR, 
AQT and RACE were actually chosen. (For more details on the construction of trees see 
Breiman ot al., 1984.) Results indicate that the best predictors consist of FAR, AQT and 
RACE (Figure 2). In this and other tree pictures, ovals represent non-terminal nodes and 
rectangles, terminal ones. Each node is labeled with the proportion or average in that node 
— in this case, the proportion of aviators undergoing attrition. Beneath each node the 


number of aviators falling into that node is given. 


2612 


FAR<6.5 
FAR>6.5 






1400 


AQT <4.5 FAR<8.5 


AQT>4.5. FAR>8.5 
228 984 tf 6 


RACE: Black, Othen 
RACE:White 


Figure 2. Reduced Aviator CURRSTAT Tree, with FAR, AQT 
and RACE as significant predictors of completion and attrition. 
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The above figure indicates that those individuals who score six or less on the FAR have a 
24.4% attrition rate, while those who score 7 or above have a 12.3% attrition rate. Further, 
individuals who have a score or 4 or less, given that they have scored a 6 or less on the 
FAR have a 38.6% attrition rate. Finally, those individuals who are “Black” or “Other” in 
the RACE category are predicted to have a 54.8% attrition rate, given that they have a 
score of 6 or less on the FAR and a 4 or less on the AQT. The tree model indicates that the 
Navy should focus efforts to recruit individuals with higher FAR and AQT scores. 

After CART optimal tree determination, a Logit model was formed using FAR, 
AQT, and RACE, along with MAJOR and COLLEGE. A full model was created using 
single predictors and their two-level interactions. Following creation of this model, a step 


function using the AIC was utilized to determine the optimal reduced model. Final model 





formulation is: 


L, = B. + B.FAR + B,AOT + B,RACEOther + B,RACEWhite + B,MAJOR1+ 8,MAJOR 2 
+ B,COLLEGE 1+ B,COLLEGE 2+ B,FAR : AOT + B,,FAR : COLLEGE 1+ 8,,FAR : COLLEGE 2 


Results fon the Logistic regression model indicate that engineering majors perform 
significantly better than individuals who hold non-technical degrees (MAJOR2 (t(2600)= 
3.39, p < 0.001). Individuals who attended the Naval Academy also performed better than 
those who attended non-ranked civilian institutions (COLLEGE2 (t(2600) = 2.75, p < 
0.01). In addition, FAR (t(2600) = 5.51, p < 0.001), AQT (t(2600) = 2.48, p < 0.01) and 
RACE White (t(2600) = 4.03, p < 0.001) are significant. In this model, “White” individuals 
perform significantly better than their “Black” counterparts. In addition to the single 


variable predictors above, a significant negative interaction was observed between FAR 


and AQT (t(2600) = -3.14, p < 0.01). Finally, an interesting interaction was observed 
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between individuals who attended the Naval Academy and how they performed on the 
FAR. Results indicate that each point increase on the FAR decreases the log-odds by 0.165 


(t(2600) = -2.25, p < 0.05). Reasons for this are unclear. Table 2 gives a listing of the 
above independent variables, their coefficient estimates B,, their resulting t-statistics and 


finally, their level of significance. 


LOGIT Std. Error tvalue Significance 

(Intercept) -3.8352 0.8478 -4.52 NS 
FAR 0.7527 0.1366 5.51 mais 
AQT 0.3862 0.1557 2.48 = 
RACEOther 0.1256 0.2807 0.48 NS 
RACEWhite 0.8482 0.2107 4.03 ee 
MAJOR1 0.1259 0.1696 0.74 NS 
MAJOR2 0.4223 0.1246 3.39 a 
COLLEGE1 -0.5809 0.6003 -0.97 NS 
COLLEGE2 1.2965 0.4712 2.75 sd 
FAR:AQT -0.0710 0.0226 -3.14 eis 
FARCOLLEGE1 0.0753 0.0888 0.848 NS 
FARCOLLEGE2 0.1651 0.0735 -2.25 





*** Significant at the alpha = 0.001 level 
** Significant at the alpha = 0.01 level 
* Significant at the alpha = 0.05 level 


NS Not Significant 


Table 2. Logit regression results for Aviator CURRSTAT 
model. 


The results of the Logit model can be used by the Naval Recruiting Command to set 
quotas for student naval aviator billets. Each year, the Chief of Naval Operations Aviation 
Air Warfare identifies the number of student aviator billets required to maintain aviation 
military readiness in the future. There are fewer billets available due to the military budget 
reduction, thus manpower analysts must maintain high retention rates for each aviation 
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year group or there will be lisa shortages several years later. An aviation manpower 
analyst can use this Logit model to predict whether or not an individual will successfully 
complete flight school training. By selecting individuals who have a higher probability of 
completing flight school, the Navy may not experience the current retention problems that 
has plagued the aviation community these past few years. For example, suppose a recent 
white USNA graduate with an engineering degree scored a six on both the FAR and on the 
AQT. Using these characteristics, a resulting Logit for this individual can be calculated 
using the model formulation below: 


L, = By + B,FAR+ B,AOT + B,RACEOther+ B,RACEWhite+ B.MAJOR\ + 8, MAJOR2 
+ B,COLLEGH + B,COLLEGE2 + B,FAR: AOT + B,,FAR : COLLEGE) + B,,FAR: COLLEGE2 


LE = ~—3.8352+0.7527* 6 + 0.3862* 6 + 0.1256* 0 + 0.8482 *1+ 0.1259* 0+ 0.4223*1 
+ (0.5809) * 0 + 1.2965 * 1+ (—0.0710) * 6 * 6 + 0.07530* 0 + (0.1651) * 6 
= 2.0186 


Performing the inverse transformation: 


A ] 1 
ear preD ~ D4 on 20186 = 0.883 





From this, it is predicted that this individual would have a success probability of 88%. 
On the other hand, if a recent black student who graduated from a non-ranked institution 
with a non-technical degree had scored a 4 on the AQT and a 5 on the FAR was 


considered. Calculating the appropriate Logit: 
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L, = By + BFAR+ B, AQT + B,RACEOther+ B,RACEWhite+ B, MAJOR + B, MAJOR? 
+ £,COLLEGH + B,COLLEGE2 + B,FAR: AOT + 8, ,FAR: COLLEGE + 8 FAR: COLLEGE 


L = —3.8352 + 0.7527* 5 + 0.3862 * 4 + 0.1256* 0 + 0.8482 * 0 + 0.1259* 0+ 0.4223*0 
+ (-0.5809) * 0+ 1.2965 * 0 + (-0.0710) *5 * 4+ 0.07530* 0 + (0.1651) * 0 
= 0.5310 


Performing the inverse transformation: 


pet __ | 89 6097 


Lee ° 





This individual’s predicted success is 63%. 

The next analysis of the student aviator data dealt with the results of the pilots who 
ultimately completed training on their first composite grade from aviation flight school. 
Model formulation was the same as above, but this time using the standardized score, 
NASCSSC, as the dependent variable. Model formulation is: 

NASCSSC ~ SEX + AQT + FAR + RACE + MAJOR + ETHNIC + SOURCE + 
COLLEGE + GPA + MATH + PHYSICS 

This score is the first that student aviators receive upon admittance into the flight-training 
program. It represents a time when they are most competitive, and are striving to perform 
at their peak in order to have the best pipeline selection choice. Results for deviance 
reduction are similar to the findings above, with an optimal tree of three terminal nodes 
created. Most important are FAR and AQT. This result is not surprising, since the purpose 
of the AQT and the FAR is to predict performance in flight school during the API and 
primary phases. Individuals who score seven or less on the FAR are predicted to have a 


NASCNSS score of 51.49%, while individuals who score above seven are predicted to 
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have a score of 54.53%. Further, individuals who score five or less on the AQT, given that 
they have scored a seven or less on the FAR are predicted to have a score of 49.88%. The 
trend indicates that higher FAR and AQT scores lead to higher composite scores achieved 


during API. Figure 3 shows the reduced tree model. 


2144 


FAR<7.5 
FAR>/7.5 





AQT>5.5 


Figure 3. Reduced NASCNSS tree model. 
Indicates that only FAR and AQT are significant 
predictors of API performance. 


A plot of the normalized distribution of the residuals was made for the NASCNSS 
data. Results (Figure 4) indicate highly skewed tails on both ends of the data set. A review 
of the original data indicated that some individuals had very low scores, yet had completed 
the training pipeline. Similarly, some individuals had high composite scores, while their 
raw score was lower. This may be indicative of poor data collection and entry efforts and 


adversely impacts the significance of this finding. 
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Figure 4. Standard normal plot of quantiles vs. 
residuals for the NASCNSS Model. 


Least-Squares regression on the NASCNSS model was conducted. A step function 
was utilized to determine the optimal linear model, using the AIC. The final model 
included FAR, AQT, MAJOR and COLLEGE, with a significant interaction between FAR 
and AQT. Both FAR (t(2136) = 4.87) and AQT (t(2136) = 4.66) are significant with 
resultant p-values = 0.00. Technical degreed aviators (t(2136) = 2.08, p < 0.05) and 
engineers (t(2136) = 2.53, p < 0.05) were also significant. Of note is that there is no 
significant effect from individuals that attended the USNA as compared to the baseline 
group. However, individuals who attended top ranked institutions performed significantly 
better than those who attended non-ranked universities (t(2136) = 3.04, p < 0.005). Finally, 
a high degree of correlation was evident between FAR and AQT (0.9143), leading to a 
significant interaction between these two terms (t(2136) = -2.93, p < 0.005). A possible 
explanation for this result 1s that some individuals might score high on one predictor but 
low on the other; however, this is not entirely clear from the analysis. The total variation 
explained by the model was only 12%, indicating that the model does not fit very well. A 


summary of the data output 1s displayed in table 3. 
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Value Std. Error tvalue  Pr(>|t}) 
(Intercept) 35.750 2.574 13.887 0.000 
FAR 1.740 0.357 4.870 0.000 
AQT 2.075 0.445 4.662 0.000 
MAJOR1 0.458 0.221 2.076 0.038 
MAJOR2 0.261 0.103 2.534 0.011 
COLLEGE1 0.546 0.180 3.036 0.002 
COLLEGE2 0.693 0.097 0.715 0.475 
FAR:AQT = -0.175 0.060 ~2.932 0.003 
Table 3. Least-Squares regression results for 
aviator NASCNSS model. 


All pilots, after completion of API, transition to primary flight training. In primary, 
the real education of a pilot begins. Students are introduced to the flight trainer and 
actually spend time in the cockpit learning to develop flying skills. In addition, academic 
work continues, and students receive a composite score for each part during this phase of 
the training program. The ASTB is designed to use AQT to predict academic performance, 
while the FAR is supposed to predict performance in the flight school eee of primary. 
Of interest is whether or not these characteristics hold true, and see if there are other 
significant predictors using CART. The academic model formulation is: 

PASS ~ SEX + AQT + FAR + RACE + MAJOR + ETHNIC + SOURCE + COLLEGE 
+ GPA + MATH + PHYSICS 

Analysis followed the same methodology as foe and an optimal minimal spanning tree 
with nine terminal nodes was developed. Results indicate that individuals who score six or 
lower and are not “White” score the lowest, with a predicted score of 43.55%. On the 
other hand, “White” individuals who score below a seven on the FAR have scores ranging 


from 48.28% to 57.03% based on their AQT score. Lower AQT scores are indicative of 


lower scores during the academic portion of Primary. For individuals that score a seven or 


above on the FAR and have a technical or an engineering degree, they are predicted to 
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score a 55.42% during this training phase. Further, if they score eight or higher on the 
AQT, they are predicted to finish with a 57.26% final score. Close examination of the tree 
model indicates that lower FAR and AQT scores result in lower predicted composite scores 
achieved during the academic portion of primary (Figure 5). Finally, using the normalized 
QQ plot to inspect the distribution of residuals, it 1s apparent that there is one point that 
requires further investigation (Figure 6). A review of the data set indicated that this 
individual had the highest standardized score, while his raw score was lower than that of 
many of his peers. It is believed that this is another case where faulty data entry 1s causing 


the discrepancy. 
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Figure 5. Reduced PASS tree model. Results indicate that FAR, 
RACE, MAJOR, and AQT are significant factors in tree 
construction. 
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Figure 6. Standard normal plot of quantiles vs. 
residuals for the PASS model. 


A Least Squares regression acti was again formulated using the techniques 
described above with the results shown in table 4 below. Independent variables used in 
model selection include FAR, MAJOR, AQT, RACE and COLLEGE. Notice that all 
factors are significant at the alpha = 0.05 level or better in this model. However, the total 
variation explained by the model was only 8%, indicating that while statistically 


significant, there is question about the usefulness of this result. 


Value Std. Error t value Pr(>|t}) 

(Intercept) 30.8379 1.2957 23.7997 

FAR 0.9016 0.1254 7.1904 0 
MAJOR1 1.8064 06129 2.9474 0.0032 
MAJOR2 2.6708 0.4101 6.513 0 
RACEOther 2.7437 1.3982 1.9624 0.0498 
RACEWhite 6.2315 1.056 5.901 0 
AQT 1.1012 0.1557 7.0736 0 
COLLEGE? 1.4543 0.4989 2.9149 0.0036 
COLLEGE2 1.4562 0.4397 3.3115  Q.0009 


Table 4. Least-Squares regression results for aviator 


PASS model. 
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The final analysis was conducted, similar to the methods above, on the PFSS 
dependent variable. FAR is expected to be a significant predictor of these scores, based 
upon the validity of the ASTB. Model formulation is: 

PFSS ~ SEX + AQT + FAR + RACE + MAJOR + ETHNIC + SOURCE + COLLEGE 
+ GPA + MATH + PHYSICS 

Again, there is significant deviance reduction, with an optimal tree of nine nodes. A tree 
grown using this value resulted in FAR , RACE, COLLEGE, MAJOR, and ETHNIC being 
the important factors in this model. Individuals who score six or less on the FAR and are 
not “White” are predicted to score lowest among all groups. Non-technical majors who 
attend non-ranked colleges also fare poorly, if they score a seven on the FAR, as compared 
to their technical and engineering counterparts who fall within the same category. The 
minimal spanning tree plot is displayed below (Figure 7) along with the normalized QQ 
plot (Figure 8). 

A Least Squares regression analysis was conducted on the PFSS data using 
predictors FAR, AQT, MAJOR, ETHNIC and COLLEGE. Use of the step model removed 
AQT from consideration. Results indicate that FAR was highly significant (t(2129) = 
15.62, p = 0.00) while MAJOR did not have an effect on performance during PFSS. There 
was a significant effect of the RACE predictor, with “White” (t(2129) = 4.44, p 0 0.00) and 
“Other” (t(2129) = 2.55, p < 0.01) categories performing significantly better than “Blacks.” 
In addition, the USNA graduates fare better than their non-ranked school counterparts at 
the alpha = 0.10 level. Significant interactions were present. Individuals who attended the 
USNA that held technical or engineering degrees performed better than their non-technical 
counterparts. Finally, it is important to note that these predictors explained only 16% of 
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Figure 7. Reduced PFSS tree model. Results indicate that FAR, 
RACE, MAJOR, and AQT are significant factors in tree 
construction. 


34 





residuals{Pilot pfss tr.9) 





Quantiles of Standard Normal 


Figure 8. Standard normal plot of quantiles vs. 
residual for PFSS model. Some outliers are 
present, as indicated by the skewed plot tails. 


the total variation in the data set as evidenced by the value of R” that was calculated for this 


regression model. Table 5 displays the output for this model. 





Value Std. Error t value Pr( > |t} ) 
(Intercept) 28.6916 2.2066 13.0025 0 
FAR 1.8821 0.1205 15.6162 0 
MAJOR1 -0.5652 1.0826 -0.5221 0.6017 
MAJOR2 0.7012 06531 1.0737 0.2831 
RACEOther 43549 1.7067 2.5517 0.0108 
RACEWhite 4.7542 1.0705 4.441 0 
COLLEGE’ 0.8577 0.6979 1.2289 0.2192 
COLLEGE2 1.2226 06749 1.8114 0.0702 
ETHNICHisp -0.1234 1.8008 -0.0685 0.9454 
ETHNICNone 3.9332 1.8427 2.1345 0.0329 
ETHNICOther 2.6099 1.8536 1.408 0.1593 
MAJOR1COLLEGE1 2.3856 1.9485 1.2243 0.221 
MAJOR2COLLEGE1 ~~ -1.0833 1.0511 -1.0306 0.3028 
MAJORICOLLEGE2 2.4013 1.3951 1.7212 0.0854 
MAJOR2COLLEGE2 1.6045 0.9298 1.7256 0.0846 


Table 5. Least-Squares regression results for aviator PFSS 


model. 
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V. DISCUSSION 
A. SUMMARY 

The present policy of the Aviator Recruiting Command is to recruit those 
iadividuals most likely to succeed in the flight-training program, based upon their ASTB 
scores and physical attributes. In addition, emphasis is placed on recruiting individuals 
having “technical” undergraduate majors. This follows the “Rickover Hypothesis,” that 
individuals with technical degrees make better officers than their non-technically educated 
counterparts. The recruiting focus is supposed to be placed on individuals with technical 
degrees; however, it is apparent that this precept is not followed because of the nearly equal 
proportions of aviators with non-technical majors and those with technical or engineering 
majors combined. This distribution provides the ability to make a direct comparison 
between the performances of each group. The evidence indicates that aviators with 
engineering degrees have a greater propensity for completing the training pipeline than 
their technical and non-technical counterparts. This echoes the conclusion found by 
Woelper (1998), who determined that SO’s with engineering degrees had higher 
completion rates than their non-engineering counterparts in the nuclear power training 
process. However, he also indicated that no effect on performance was observed after 
completion of training, indicating that the effect of undergraduate major diminishes as one 
progresses upward in his or her Naval Career (Woelper, 1998). 

Many factors affect pilot performance during the training process; however, they 
are numerous and not well defined. Initial review of the above results seems to indicate 
that MAJOR and COLLEGE are significant factors in pilot development; however, this 
conclusion must be viewed with caution. Although the explanatory variables for each 
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performance measure were statistically significant, one must ask whether or not these 
variables account for all the variation observed by the model formulation. Because residual 
deviance was very high in the CART models, and the corresponding R? value was 
extremely low in all of the least squares regression models, inferences about this result are 
difficult to make. However, certain observations can be made. 

The first observation to consider is MAJOR. The data suggest that individuals with 
engineering degrees perform better during API and in the academic portion of Primary 
flight training, as indicated by significant findings in these areas. On the other hand, the 
effect of holding a “technical” degree was not significant during the flight portion of 
Primary. This indicates that while a strong educational background in a technical field 
increases the likelihood of having higher academic grades and less likelihood of 
withdrawal from the flight-training program, there is little relationship between this degree 
and flight school performance in the air. 

The second observation deals with the importance of COLLEGE. In all but the 
NASCNSS model, USNA attendance was an important predictor of success, whether 
dealing with completion rates, or when comparing grades. It can be hypothesized that 
individuals who attended the Naval Academy have better preparation than their NROTC 
and OCS counterparts due to the unique experience they receive through attendance and 
participation in the military institution. In short, they perform better because they are more 
disciplined and require less time to acclimate to the military surroundings than their 
counterparts, resulting in greater attention to the primary task at hand. It is important to 
note that SOURCE was not a factor named in any of the CART models, indicating that 
there is not much difference when approached from this angle. However, evidence 
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suggests that individuals who attend top-ranked institutions and those who attend the 
USNA fare better than their counterparts, regardless of whether or not they entered through 
OCS or NROTC. 

Finally, both the AQT and FAR measures were identified in the CART models, 
leading to the belief that they are performing their job as predictors of success in the 
aviation program. The results of this study seem to further validate these measures of 
performance and suggest that the ASTB be retained as an initial screening tool for aviators. 

Several issues and problems with the above conclusions need to be discussed. First 
is the issue of data accuracy and completeness. In many instances, large blocks of the data 
set had to be excluded due to missing data or other inaccuracies in the data files. For 
instance, 795 members of the NAMRL data set were unable to be matched by social 
security number in the OMF data files, even though they were listed as being naval 
officers. Numerous other instances occurred where individuals were listed as completing 
the flight-training pipeline, but did not have NASCNSS, PFSS, or PASS grades entered. 
Another problem was that some individuals were listed as having attended the USNA, as 
identified by COLLEGE, yet these same individuals were listed as having commissioning 
source of OCS or NROTC. All of these individuals were removed from consideration 
during analysis. 

Another important issue is the question of sample size. In statistical analysis, a 
large sample size enables one to detect even small differences between measures of 
performance, based on explanatory variables. However, judgment must be made to 
determine if statistically significant results are of practical use. These results must be 
viewed with this in mind. A population of 2612 pilots provides ample room to find even 
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minute differences between groups of people. Further, the population from which the 
results were derived must be representative of the set of all aviators who attended flight 
school. In conclusion, it is believed that differences do exist between pilots, based upon 
their academic background. However, the practical necessity of recruiting pilots who meet 
the minimum standards to maintain required manning levels requires focus on those that 
might not have attributes that are strong predictors of success as determined from this 
analysis. 

A third issue deals with the absence of data reflecting actual performance of an 
individual during his or her undergraduate education. The APC is supposed to provide 
information regarding overall academic performance, in addition to performance in 
mathematics and physics. Only 40% of the individuals in the data set held the APC. The 
APC is computed by the Naval Postgraduate School (NPS), usually within three years from 
commissioning. However, individuals must ensure that college transcripts are forwarded to 
the school for computation. This score is required for acceptance into NPS and for 
consideration of acceptance into other graduate school programs. An individual who does 
not desire to pursue a graduate education might not request that their transcripts be 
forwarded. It is possible that a significant number of individuals who lack this code might 
have lower grades, and therefore consider themselves ineligible for graduate education. 
The APC and its sub-scores of GPA, MATH and PHYSICS were not utilized in final 
model formulation; however, with a more complete data set, this information could have 
been used to develop a more accurate picture of flight school performance. 

Most important, however, is the issue of the determination of completion and 
attrition. The data sets provided did not provide reasons for attrition, necessitating all those 


AQ 





who withdraw be included in the model. Identification of those who attrite for medical 
reasons, or reasons not related to individual performance, would hopefully provide more 
useful data for analysis. This would limit the analysis to that of those individuals who had 
difficulty in performing academic or flight-related tasks and enable a more precise 
identification of factors related to successful flight school completion. Nevertheless, the 
models do indicate a relationship between AQT and FAR and flight school performance, 
indicating that these measures are performing as designed. The factors of MAJOR, 
COLLEGE, and RACE also provide predictive power. Even without knowing the cause of 
attrition, inferences can be made on an individual’s performance, based upon the factors 
listed above. This model does not attempt to predict attrition, however. It merely identifies 
factors that indicate successful flight school completion and, using the CART 
methodology, classifies individuals into groups with common characteristics. 
B. RECOMMENDATIONS 

The present recruiting policy for naval aviators is to place emphasis on recruiting 
individuals with technical degrees. This policy is sound, given the above results. There are 
limitations to this, as indicated by the number of aviators with non-technical degrees who 
are already in the fleet. It is impossible, in today’s difficult recruiting environment, to find 
only those individuals who have engineering degrees and otherwise meet all entrance 
specifications. The CART and Logit models should be utilized to compare traits of 
individuals, recognizing that each of the significant factors in this nig is associated with 
the individual’s predicted success likelihood. Those individuals who have characteristics 


associated with higher success probability or to higher predicted composite scores should 
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be recruited, if possible. Focus should be given to those individuals who are predicted to 
perform the best in the training program, as described below. 

There are several factors that seem to provide consistent improvements throughout 
this analysis. These include AQT, FAR, RACE, MAJOR and COLLEGE. Higher ASTB 
scores, which predict success likelihood in both the academic and the flight school portions 
of the training pipeline, result in higher predicted completion rates and higher composite 
scores. Individuals who have technical degrees, or who attend the Naval Academy also 
fare better than their counterparts. Neither of these results is surprising. Individuals who 
are White have higher completion rates, and higher composite scores, too. This result was 
unanticipated and the reasons for these differences are not clear. Perhaps it reflects the 
contribution of other, unmeasured attributes of the aviators; possibly it reflects cultural 
biases in the tests themselves. This thesis does not recommend against selection of racial 
minorities; however it does suggest that the selection process can take into account an 
“ individual’s background and use the Logit or CART model to estimate success probability. 
Students with low estimated probabilities are natural targets for intervention strategies 
designed to improve retention and thereby preserve the Navy’s investment. 

There are some areas that need to be analyzed further. First, this study only looks at 
pilots, and not at Naval Flight Officers. However, during the analysis, some statistical 
work was done relating to this group of individuals in which results similar to those found 
above were obtained. A more detailed analysis should be conducted before conclusions 
about the NFO community are reached. 

Second, greater emphasis needs to be placed to ensure accuracy and completeness 


of data entry. All statistical analysis can only be as good as the data it models. It is 
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therefore recommended that the data collection process be reviewed and improvements 
made in order to improve the process of statistical analysis. 

Individual pilots maintain training folders; however, these folders are kept locally at 
each command and are not compiled into a sentra database. There is no methodology for 
tracking aviator fleet performance, such as safety check flights, re-qualification exams, 
upgrades and performance qualifications. It is recommended that a centralized 
methodology for tracking the performance and qualifications of pilots be developed, with 
the ability to measure trends in performance after the qualified pilot leaves flight school. 
This database would provide a useful source for measuring trends in pilot performance 
throughout the fleet, and give a way of tracking trends occurring in the pilot community. 
In addition, it would provide managers a possible way to identify problem aviators before 
they create a crisis. 

Most important, however, is the need to follow up on the individuals included in 
this study for the purpose of tracking them throughout their careers. The pilots included in 
this study have only begun their obligated service requirement, and have therefore not 
reached major career decision points. Identification of trends in retention and promotion, 
as related to flight school experience, is an area of study that should be examined. At issue 
is whether or not those individuals who performed well in flight school, based upon their 
composite scores, along with their demographic background, display measurable trends 
that affect early promotion and retention likelihood. In short, can the results of this study 
be extended to the career of a pilot? A follow-up study to measure the promotion and 


retention rates for these individuals should be conducted as their career progresses. 
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APPENDIX A. COMBINED DATA FILE DESCRIPTIONS 


Data Typeleodelnotes 


| Social Security Number Type: Character 


anal Current Status Type: Factor 
Code: ATTRITION 
COMPLETION 


Code: M - Male 
F - Female 


Aviation Qualification Test | Type: Integer 













Code: 0-9 Test Score Results 
FAR Flight Aptitude Rating Type: Integer 


Code: 0-9 Test Score Results 


NASCNSS NASC Standard Score (%) Type: Double 
FINALCOM Final Composite Score Type: Double 


Race Type: Factor 
Code: black 
white 
other 


MAJOR Undergraduate Major Type: Factor 





Code: 0 - Non-Technical 
1 - Technical 
2 - Engineer 


Note: Derived from OPNAVINST 1530 





Factor 






Type: 









Code: Asian 
Hisp 
None 
Other 








ETHNIC Ethnic Group 


SOURCE Commissioning Source 


MATH 
PHYSICS Physics GPA 


COLLEGE Rating of College Attended : Factor 







: Factor 










: OCS 
NROTC 
USNA 









Grade Point Average : Factor 





: 0 - Less than 3.20 GPA 
1 - Greater than 3.20 GPA 








Mathematics GPA : Factor 








: 0 - Calculus Grade B or lower 
1 - Calculus Grade B+ or better 







: Factor 






Code: Q - Physics Grade B or lower 
1 - Physics Grade B+ or higher 





: QO - Attended Non-rated School 
1 - Attended top ranked National 
or Liberal Arts College 


2 - Attended USNA 


NASCRAW NASC Raw Score (%) Type: Double 


PFG Primary Raw Flight Grade Type: Double 


PAG Primary Raw Academic | Type: Double 
pe |Get ME OO 
PFSS Primary Flight Standard | Type: Double 
Se 
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— | 





Primary Academic Standard | Type: Double 
— is (%) 


PIPELINE Aircraft Type Type: Factor 
Code: Helo 
Jet 
Prop 
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APPENDIX B. 


attritions and completions. 
SSN CURRSTAT SE 
Length: 2612 ATTR: 468 Es 
Class: AsIs COMP:2144 M: 
Mode:character 
AQT FAR 
Mins+3.000 Mine 23000 
ist. Oux 25>. G00 ist Ou.<5 7000 1 
Median:6.000 Median:7.000 
Mean:6.072 Mean:6.777 
S¥C: Oli 7.000 Sra-00..2°6.000 5 
Max.:9.000 Max.:9.000 
RACE MAJOR ETHNIC 
Black: 122 02-1322. Asian: 59 
Other: 121 1: 304 Hisp: 118 
White:2369 2: 987 None:1610 
Other: 825 
PHYSICS COLLEGE PIPELINE 
21576 Oe 1042 NPH:776 
Le 305 L<. - 575 NPM: 607 
Os. 733 2: 991 NPI: 382 
NPO:241 
NPA: 204 
NPT:148 
(Other) :254 
PAG PFSS 
Min.: 0.00 Min.: 0.00 


lst QOu.: 90.67 
Median: 94.00 
Mean: 87.53 
Sra. Ouse: 96.27 
Max.:100.00 
NA*s: 10.00 


ist QOu.241.00 
Median: 49.00 
Mean:45.85 
std. Ou<¢56..00 
Max.:99.00 
NA's:42.00 


SUMMARY OF PILOT DATA 


Data below include all members of the student aviator data set, including both 


x COMMDATE 
150 Min.:10960 
2462 Pst 00472218640 
Median:12380 
Mean:12420 
Sra: Ou. 213010 
Max.:14230 
NASCNSS FINALCOM 
Min:.2 0'..00 Min.: 0.0 
st O0<2 47.00 lst. Ou. 2162.0 
Median:53.00 Median:193.0 
Mean:50.22 Mean:166.7 
rd 0Ou-.*57 .00 Sra Ouse 215.4 
Max.:97.83 Max.:304.0 
SOURCE GPA MATH 
NROTC:1235 nel ae As 21576 
OCS: 3686 Le - 3G des 220 
USNA: 991 Oe: 720 O: 816 
NASCRAW PFG 
Mans 2.0.00 Min.:0.000 
TSt Onset o7.62 ist. Ou.23.034 
Median: 91.00 Median:3.063 
Mean:77.58 Mean:2.828 
3rd Qu.:95.00 Srd Ou.73.088 
Max.:99.99 Max.:3.991 
NA's:9.000 
PASS PIPELINE 
Min.: 0.0 56 
ist Ou.:44.0 helo: 776 
Median:52.0 Jet 21029 
Mean:48.1 prop: fol 
3rd Ou.?59.0 
Max.:95.0 
NA's: 42.0 
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Data below include all members of the student aviator data set who have completed 


ON 
the entire training pipeline. 
SSN CURRSTAT SEX COMMDATE 
Length: 2144 ATTR: 0 F: 120 Min.:10960 
Class: AsIs COMP:2144 M:2024 Let: Ou. 112620 
Mode:character Median:12200 
Mean:12340 
3rd O00 2212930 
Max.:13870 
AQT FAR NASCNSS FINALCOM 
Min.=:3.00 Min.:3.000 Mint Z0 200 Min.: 90.0 
TSE OG. 45.00 ‘st. OW. 262000 lst Quis49.00 1st. Ove £13850 
Median: 6.00 Median: 7.000 Median: 53.00 Median:200.1 
Mean:6.13 Mean: 6.928 Mean:52.75 Mean:201.2 
3rd OuU~.2 7.00 3rd Qu.:9.000 3rd. -0.2:57-.00 ord. Oe? 220-6 
Max.:9.00 Max.:9.000 Max.:97.83 Max.:304.0 
RACE MAJOR ETHNIC SOURCE GPA MATH 
Black: 71 0:1036 Asian: 38 NROTC:1024 21299 21299 
Other: 81 1: 247 Hisp: 89 OCS: 289 13252 cs. Leo 
White:1992 2 Oo None:1349 USNA: 831 0: 593 O: 660 
Other: 668 
PHYSICS COLLEGE PIPELINE NASCRAW PFG 
71299 D283 7 NPH: 733 Mains 0.00 Min.:0.000 
1a: 249 12276 NPM:571 St -Ouet56 455 LSt. Os =32046 
O: 596 26st NPJ: 360 Median: 92.00 Median:3.069 
NPA: 190 Mean:80.05 Mean:3.055 
NPT:138 Sea 0055 95.5 3rd. Ou.23.092 
NPE:137 Max.:99.99 Max.:3.991 
(Other): i5 
PAG PESS PASS PIPELINE 
Mines °«Q200 Masy.2 17500 Min.:17.0 helo:733 
lst Qu.: 91.40 Ist Ou.:44.00 Ist Qu.:46.0 jet:703 
Median: 94.33 Median:51.00 Median:53.0 prop: 706 
Mean: 92.63 Mean:50.83 Mean:51.9 
3ra Oux.<: 96.40 376. "OUs797-00 Sra: O00.259.0 
Max.:100.00 Max.:99.00 Max.:95.0 
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